{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "93ae9bad-b8cc-43de-ba7d-387e0155674c",
   "metadata": {},
   "source": [
    "# Building a Natively Multimodal RAG Pipeline (over a Slide Deck)\n",
    "\n",
    "<a href=\"https://colab.research.google.com/github/run-llama/llama_cloud_services/blob/main/examples/parse/multimodal/multimodal_rag_slide_deck.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n",
    "\n",
    "In this cookbook we show you how to build a multimodal RAG pipeline over a slide deck, with text, tables, images, diagrams, and complex layouts.\n",
    "\n",
    "A gap of text-based RAG is that they struggle with purely text-based representations of complex documents. For instance, if a page contains a lot of images and diagrams, a text parser would need to rely on raw OCR to extract out text. You can also use a multimodal model (e.g. gpt-4o and up) to do text extraction, but this is inherently a lossy conversion.\n",
    "\n",
    "Instead a **native multimodal pipeline** stores both a text and image representation of a document chunk. They are indexed via embeddings (text or image), and during synthesis both text and image are directly fed to the multimodal model for synthesis.\n",
    "\n",
    "This can have the following advantages:\n",
    "- **Robustness**: This solution is more robust than a pure text or even a pure image-based approach. In a pure text RAG approach, the parsing piece can be lossy. In a pure image-based approach, multimodal OCR is not perfect and may lose out against text parsing for text-heavy documents.\n",
    "- **Cost Optimization**: You may choose to dynamically include text-only, or text + image depending on the content of the page.\n",
    "\n",
    "Status:\n",
    "| Last Executed | Version | State      |\n",
    "|---------------|---------|------------|\n",
    "| Aug-20-2025   | 0.6.61  | Maintained |\n",
    "\n",
    "![mm_rag_diagram](./multimodal_rag_slide_deck_img.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "54e8d9a7-5036-4d32-818f-00b2e888521f",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "73542086",
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install llama-cloud-services \"llama-index>=0.13.0<0.14.0\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a4518afd",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "os.environ[\"OPENAI_API_KEY\"] = \"sk-\"\n",
    "os.environ[\"LLAMA_CLOUD_API_KEY\"] = \"llx-...\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "225c5556-a789-4386-a1ee-cce01dbeb6cf",
   "metadata": {},
   "source": [
    "### (Optional) Setup Observability\n",
    "\n",
    "We setup an integration with LlamaTrace (integration with Arize).\n",
    "\n",
    "If you haven't already done so, make sure to create an account here: https://llamatrace.com/login. Then create an API key and put it in the `PHOENIX_API_KEY` variable below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0eabee1f-290a-4c85-b362-54f45c8559ae",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install -U llama-index-callbacks-arize-phoenix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aaeb245c-730b-4c34-ad68-708fdde0e6cb",
   "metadata": {},
   "outputs": [],
   "source": [
    "# setup Arize Phoenix for logging/observability\n",
    "import llama_index.core\n",
    "import os\n",
    "\n",
    "PHOENIX_API_KEY = \"<PHOENIX_API_KEY>\"\n",
    "os.environ[\"OTEL_EXPORTER_OTLP_HEADERS\"] = f\"api_key={PHOENIX_API_KEY}\"\n",
    "llama_index.core.set_global_handler(\n",
    "    \"arize_phoenix\", endpoint=\"https://llamatrace.com/v1/traces\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fbb362db-b1b1-4eea-be1a-b1f78b0779d7",
   "metadata": {},
   "source": [
    "### Load Data\n",
    "\n",
    "Here we load the [Conoco Phillips 2023 investor meeting slide deck](https://static.conocophillips.com/files/2023-conocophillips-aim-presentation.pdf)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8bce3407-a7d2-47e8-9eaf-ab297a94750c",
   "metadata": {},
   "outputs": [],
   "source": [
    "!mkdir data\n",
    "!mkdir data_images\n",
    "!wget \"https://static.conocophillips.com/files/2023-conocophillips-aim-presentation.pdf\" -O data/conocophillips.pdf"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "246ba6b0-51af-42f9-b1b2-8d3e721ef782",
   "metadata": {},
   "source": [
    "### Model Setup\n",
    "\n",
    "Setup models that will be used for downstream orchestration."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "16e2071d-bbc2-4707-8ae7-cb4e1fecafd3",
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.core import Settings\n",
    "from llama_index.llms.openai import OpenAI\n",
    "from llama_index.embeddings.openai import OpenAIEmbedding\n",
    "\n",
    "embed_model = OpenAIEmbedding(model=\"text-embedding-3-large\")\n",
    "llm = OpenAI(model=\"gpt-5-mini\")\n",
    "\n",
    "Settings.embed_model = embed_model\n",
    "Settings.llm = llm"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e3f6416f-f580-4722-aaa9-7f3500408547",
   "metadata": {},
   "source": [
    "## Use LlamaParse to Parse Text and Images\n",
    "\n",
    "In this example, use LlamaParse to parse both the text and images from the document."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "570089e5-238a-4dcc-af65-96e7393c2b4d",
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_cloud_services import LlamaParse\n",
    "\n",
    "\n",
    "parser = LlamaParse(\n",
    "    parse_mode=\"parse_page_with_agent\",\n",
    "    model=\"openai-gpt-4-1-mini\",\n",
    "    high_res_ocr=True,\n",
    "    adaptive_long_table=True,\n",
    "    outlined_table_extraction=True,\n",
    "    output_tables_as_HTML=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ef82a985-4088-4bb7-9a21-0318e1b9207d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Started parsing the file under job_id 2cf07879-5bdb-4dca-9a07-001b2a07727e\n",
      "."
     ]
    }
   ],
   "source": [
    "results = await parser.aparse(\"data/conocophillips.pdf\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5318fb7b-fe6a-4a8a-b82e-4ed7b4512c37",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "# Commitment to Disciplined Reinvestment Rate\n",
      "\n",
      "<table>\n",
      "<thead>\n",
      "<tr>\n",
      "  <th>Industry Growth Focus</th>\n",
      "  <th>ConocoPhillips Strategy Reset</th>\n",
      "  <th>Disciplined Reinvestment Rate is the Foundation for Superior Returns <br> <b>on and of</b> Capital, while Driving Durable CFO Growth</th>\n",
      "</tr>\n",
      "</thead>\n",
      "<tbody>\n",
      "<tr>\n",
      "  <td style=\"text-align:center;\">&gt;100%<br>Reinvestment Rate</td>\n",
      "  <td style=\"text-align:center;\">&lt;60%<br>Reinvestment Rate</td>\n",
      "  <td style=\"text-align:center; font-weight:bold; color:#0055ff;\">\n",
      "    ~50%<br>10-Year Reinvestment Rate<br><br>\n",
      "    ~6%<br>CFO CAGR 2024-2032<br><br>\n",
      "    at $60/BBL WTI<br>Mid-Cycle Planning Price\n",
      "  </td>\n",
      "</tr>\n",
      "<tr>\n",
      "  <td>\n",
      "    <div style=\"height:150px; width:50px; background-color:#b0b0b0; margin: 0 auto; position:relative;\">\n",
      "      <div style=\"position:absolute; bottom:0; width:100%; height:105%; background-color:#b0b0b0;\"></div>\n",
      "      <div style=\"position:absolute; bottom:0; width:100%; text-align:center; color:#fff; font-weight:bold;\">~$75/BBL<br>WTI Average</div>\n",
      "    </div>\n",
      "  </td>\n",
      "  <td>\n",
      "    <div style=\"height:150px; width:50px; background-color:#b0b0b0; margin: 0 auto; position:relative;\">\n",
      "      <div style=\"position:absolute; bottom:0; width:100%; height:56%; background-color:#b0b0b0;\"></div>\n",
      "      <div style=\"position:absolute; bottom:0; width:100%; text-align:center; color:#fff; font-weight:bold;\">~$63/BBL<br>WTI Average</div>\n",
      "    </div>\n",
      "  </td>\n",
      "  <td>\n",
      "    \n",
      "\n",
      "<table>\n",
      "      <thead>\n",
      "        <tr>\n",
      "          <th>Year</th>\n",
      "          <th>Reinvestment Rate at $60/BBL WTI</th>\n",
      "          <th>Reinvestment Rate at $80/BBL WTI</th>\n",
      "        </tr>\n",
      "      </thead>\n",
      "      <tbody>\n",
      "        <tr>\n",
      "          <td>2023E</td>\n",
      "          <td style=\"background-color:#3399ff; color:#fff; text-align:center;\">~50%</td>\n",
      "          <td></td>\n",
      "        </tr>\n",
      "<tr>\n",
      "          <td>2024-2028</td>\n",
      "          <td style=\"background-color:#0033cc; color:#fff; text-align:center;\">~55%</td>\n",
      "          <td style=\"border-top: 2px dashed #3399ff; text-align:center;\">at $80/BBL WTI</td>\n",
      "        </tr>\n",
      "<tr>\n",
      "          <td>2029-2032</td>\n",
      "          <td style=\"background-color:#0033cc; color:#fff; text-align:center;\">~38%</td>\n",
      "          <td style=\"border-top: 2px dashed #3399ff; text-align:center;\">at $80/BBL WTI</td>\n",
      "        </tr>\n",
      "      </tbody>\n",
      "    </table>\n",
      "\n",
      "  </td>\n",
      "</tr>\n",
      "<tr>\n",
      "  <td colspan=\"3\" style=\"text-align:center; font-size:0.8em; color:#666;\">\n",
      "    Historic Reinvestment Rate (gray) | Reinvestment Rate at $60/BBL WTI (blue solid) | Reinvestment Rate at $80/BBL WTI (blue dashed)\n",
      "  </td>\n",
      "</tr>\n",
      "<tr>\n",
      "  <td colspan=\"3\" style=\"font-size:0.75em; color:#999; padding-top:10px;\">\n",
      "    Reinvestment rate and cash from operations (CFO) are non-GAAP measures. Definitions and reconciliations are included in the Appendix.\n",
      "  </td>\n",
      "</tr>\n",
      "</tbody>\n",
      "</table>\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(results.pages[10].md)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eb5ec429",
   "metadata": {},
   "source": [
    "We can download the page screenshots directly, and we can use them as context later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "27773ef0",
   "metadata": {},
   "outputs": [],
   "source": [
    "image_nodes = await results.aget_image_nodes(\n",
    "    include_object_images=False,\n",
    "    include_screenshot_images=True,\n",
    "    image_download_dir=\"./slide_images\",\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d0ea7a69",
   "metadata": {},
   "outputs": [],
   "source": [
    "text_nodes = results.get_markdown_nodes(split_by_page=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd3e098b-0606-4429-b48d-d4fe0140fc0e",
   "metadata": {},
   "source": [
    "## Build Multimodal Index\n",
    "\n",
    "In this section we build the multimodal index over the parsed deck. \n",
    "\n",
    "We do this by creating **text** nodes from the document that contain metadata referencing the original image path.\n",
    "\n",
    "In this example we're indexing the text node for retrieval. The text node has a reference to both the parsed text as well as the image screenshot."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3aae2dee-9d85-4604-8a51-705d4db527f7",
   "metadata": {},
   "source": [
    "#### Get Text Nodes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "18c24174-05ce-417f-8dd2-79c3f375db03",
   "metadata": {},
   "outputs": [],
   "source": [
    "for text_node, image_node in zip(text_nodes, image_nodes):\n",
    "    text_node.metadata[\"image_path\"] = image_node.image_path"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8e331dfe-a627-4e23-8c57-70ab1d9342e4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "page_number: 1\n",
      "file_name: data/conocophillips.pdf\n",
      "image_path: slide_images/page_1.jpg\n",
      "\n",
      "\n",
      "# ConocoPhillips\n",
      "\n",
      "## 2023 Analyst & Investor Meeting\n"
     ]
    }
   ],
   "source": [
    "print(text_nodes[0].get_content(metadata_mode=\"all\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4f404f56-db1e-4ed7-9ba1-ead763546348",
   "metadata": {},
   "source": [
    "#### Build Index\n",
    "\n",
    "Once the text nodes are ready, we feed into our vector store index abstraction, which will index these nodes into a simple in-memory vector store (of course, you should definitely check out our 40+ vector store integrations!)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6ea53c31-0e38-421c-8d9b-0e3adaa1677e",
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.core import VectorStoreIndex\n",
    "\n",
    "index = VectorStoreIndex(nodes=text_nodes)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f0e33a4-9422-498d-87ee-d917bdf74d80",
   "metadata": {},
   "source": [
    "## Build Multimodal Query Engine\n",
    "\n",
    "We now use LlamaIndex abstractions to build a **custom query engine**. In contrast to a standard RAG query engine that will retrieve the text node and only put that into the prompt (response synthesis module), this custom query engine will also load the image document, and put both the text and image document into the response synthesis module."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "35a94be2-e289-41a6-92e4-d3cb428fb0c8",
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.core.query_engine import CustomQueryEngine\n",
    "from llama_index.core.retrievers import BaseRetriever\n",
    "from llama_index.core.schema import MetadataMode\n",
    "from llama_index.core.base.response.schema import Response\n",
    "from llama_index.core.llms import TextBlock, ImageBlock, ChatMessage\n",
    "\n",
    "\n",
    "qa_prompt_block_text = \"\"\"\\\n",
    "Below we give parsed text from slides in two different formats, as well as the image.\n",
    "\n",
    "---------------------\n",
    "{context_str}\n",
    "---------------------\n",
    "\"\"\"\n",
    "\n",
    "image_prefix_block = TextBlock(text=\"And here are the corresponding images per page\\n\")\n",
    "\n",
    "image_suffix = \"\"\"\\\n",
    "Given the context information and not prior knowledge, answer the query. Explain whether you got the answer\n",
    "from the parsed markdown or raw text or image, and if there's discrepancies, and your reasoning for the final answer.\n",
    "\n",
    "Query: {query_str}\n",
    "Answer: \"\"\"\n",
    "\n",
    "\n",
    "class MultimodalQueryEngine(CustomQueryEngine):\n",
    "    \"\"\"Custom multimodal Query Engine.\n",
    "\n",
    "    Takes in a retriever to retrieve a set of document nodes and respond using an LLM + retrieved text/images.\n",
    "\n",
    "    \"\"\"\n",
    "\n",
    "    retriever: BaseRetriever\n",
    "    llm: OpenAI\n",
    "\n",
    "    def __init__(self, **kwargs) -> None:\n",
    "        \"\"\"Initialize.\"\"\"\n",
    "        super().__init__(**kwargs)\n",
    "\n",
    "    def custom_query(self, query_str: str):\n",
    "        # retrieve text nodes\n",
    "        nodes = self.retriever.retrieve(query_str)\n",
    "        # create ImageNode items from text nodes\n",
    "        image_blocks = [\n",
    "            ImageBlock(path=n.metadata[\"image_path\"])\n",
    "            for n in nodes\n",
    "            if n.metadata.get(\"image_path\")\n",
    "        ]\n",
    "\n",
    "        # create context string from text nodes, dump into the prompt\n",
    "        context_str = \"\\n\\n\".join(\n",
    "            [r.get_content(metadata_mode=MetadataMode.LLM) for r in nodes]\n",
    "        )\n",
    "\n",
    "        formatted_msg = ChatMessage(\n",
    "            role=\"user\",\n",
    "            blocks=[\n",
    "                TextBlock(text=qa_prompt_block_text.format(context_str=context_str)),\n",
    "                image_prefix_block,\n",
    "                *image_blocks,\n",
    "                TextBlock(text=image_suffix.format(query_str=query_str)),\n",
    "            ],\n",
    "        )\n",
    "\n",
    "        # synthesize an answer from formatted text and images\n",
    "        llm_response = self.llm.chat([formatted_msg])\n",
    "\n",
    "        return Response(\n",
    "            response=str(llm_response.message.content),\n",
    "            source_nodes=nodes,\n",
    "        )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0890be59-fb12-4bb5-959b-b2d9600f7774",
   "metadata": {},
   "outputs": [],
   "source": [
    "query_engine = MultimodalQueryEngine(\n",
    "    retriever=index.as_retriever(similarity_top_k=3), llm=llm\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a92aa4f1-7501-4711-b054-f02338e54e74",
   "metadata": {},
   "source": [
    "### Define Baseline\n",
    "\n",
    "In addition, we define a \"baseline\" where we rely only on text-based indexing. Here we define an index using only the nodes that are parsed in text-mode from LlamaParse. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f6bcfbc6-4e9b-41ad-ad81-1c4245b95cd5",
   "metadata": {},
   "outputs": [],
   "source": [
    "base_index = VectorStoreIndex(nodes=text_nodes)\n",
    "base_query_engine = base_index.as_query_engine(llm=llm, similarity_top_k=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1f94ef26-0df5-4468-a156-903d686f02ce",
   "metadata": {},
   "source": [
    "## Build a Multimodal Agent\n",
    "\n",
    "Build an agent around the multimodal query engine. This gives you agent capabilities like query planning/decomposition and memory around a central QA interface."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5b7a8c5f-39fc-4d04-8c56-3642f5718437",
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.core.tools import QueryEngineTool\n",
    "from llama_index.core.agent import FunctionAgent\n",
    "\n",
    "\n",
    "vector_tool = QueryEngineTool.from_defaults(\n",
    "    query_engine=query_engine,\n",
    "    name=\"vector_tool\",\n",
    "    description=(\n",
    "        \"Useful for retrieving specific context from the data. Do NOT select if question asks for a summary of the data.\"\n",
    "    ),\n",
    ")\n",
    "agent = FunctionAgent(\n",
    "    tools=[vector_tool],\n",
    "    llm=llm,\n",
    ")\n",
    "\n",
    "from llama_index.core.workflow import Context\n",
    "\n",
    "# Context to store chat history for the session\n",
    "ctx = Context(agent)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2b4f7eb1-d247-45fa-bb41-c02fc353a22a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# define a similar agent for the baseline\n",
    "base_vector_tool = QueryEngineTool.from_defaults(\n",
    "    query_engine=base_query_engine,\n",
    "    name=\"vector_tool\",\n",
    "    description=(\n",
    "        \"Useful for retrieving specific context from the data. Do NOT select if question asks for a summary of the data.\"\n",
    "    ),\n",
    ")\n",
    "base_agent = FunctionAgent(\n",
    "    tools=[base_vector_tool],\n",
    "    llm=llm,\n",
    ")\n",
    "\n",
    "base_ctx = Context(base_agent)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2336f98b-c0a1-413a-849d-8a89bacb90b5",
   "metadata": {},
   "source": [
    "## Try out Queries\n",
    "\n",
    "Let's try out queries against these documents and compare against each other."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d78e53cf-35cb-4ef8-b03e-1b47ba15ae64",
   "metadata": {},
   "outputs": [],
   "source": [
    "query = (\n",
    "    \"Tell me about the diverse geographies where Conoco Phillips has a production base\"\n",
    ")\n",
    "\n",
    "response = await agent.run(query, ctx=ctx)\n",
    "base_response = await base_agent.run(query, ctx=base_ctx)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "355d2aa4-c26f-480e-b512-4446acbd9227",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(str(response))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d21d694b-6618-4d04-a6f6-8b0c2625f539",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(str(base_response))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
